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Summary 


1.  Define  N  to  be  the  number  of  objects  in  a  scenario.  Our  interest  is  in  values  as  high  as 
10'*  <  N  <  10®.  We  have  performed  initial  theoretical  analysis  of  “gating”  (selection 
of  candidate  track-report  pairs). 

a.  A  straightforward  brute  force  algorithm  should  scale  as  . 

b.  A  nonhierarchical  cluster  algorithm  used  in  multihypothesis  tracking  [1]  should 
scale  at  best  as  and  probably  closer  to  N"^.  The  scaling  of  this  algorithm 
depends  primarily  on  the  track  and  report  data  and  on  the  scenario.  Obtaining 

scaling  is  fortuitous  and  is  not  within  the  control  of  the  programmer. 

c.  The  worst  scaling  part  of  the  LCPFD  near-neighbor  algorithms  should  vary  as 
N  log  N. 

2.  The  calculation  of  scores  for  pairings  of  tracks  with  reports  is  the  most  expensive 
part  of  gating  for  the  brute  force  and  cluster  algorithms.  Near-neighbor  eilgorithms 
should  reduce  the  number  of  score  calculations  considerably.  This  part  of  gating 
should  increase  linearly  with  N  for  near-neighbor  algorithms,  as  for  brute  force 
and  between  N®/*  and  iV*  for  the  cluster  algorithm. 

3.  The  two  aspects  which  these  tests  will  measure  are  scaling  of  the  cost  with  N  and 
accuracy  of  the  result.  For  accuracy  two  measures  of  performance  will  be  used: 

a.  Vi  =  the  number  of  highest-valued  pairs  actually  found  divided  by  the  nmnber 
of  highest-valued  pairs  which  should  have  been  found. 

b.  ‘P2  =  “Pi  X  the  sum  of  scores  of  pairs  actually  foimd  divided  by  the  sum  of  scores 
of  pairs  which  should  have  been  found. 

4.  To  measure  scaling  with  N,  the  normalized  cost  of  each  major  step  in  the  gating 
process  for  several  values  of  N  will  be  fit  to  the  functions  jSJV®  and  N{a  log  JV  -1-  /3), 
where  a  and  P  are  constants  to  be  evaluated,  in  order  to  determine  the  most  likely 
functional  dependence.  We  will  express  cost  in  terms  of  elapsed  central  processor  time 
normalized  by  the  time  required  for  a  given  algorithm  at  the  smallest  value  of  N. 

5.  Data  set  size  requirements  differ  for  the  measurement  of  scaling  and  accuracy.  For 
scaling,  N  must  vary  at  least  from  100  to  10,000  and  perhaps  up  to  60,000,  since  the 
systems  of  interest  might  contain  as  many  as  10*  to  10®  objects.  Data  sets  covering 
the  range  of  values  100  <  N  <  1,000  appear  to  be  adequate  for  measuring  accuracy. 


6.  For  measuring  accuracy  of  the  algorithms  and  to  cover  the  various  situations  which 
affect  scaling  of  the  cluster  algorithm,  the  scenEirios  should  cover  a  range  of  cluster 
sizes.  Clusters  shovild  vary  from  an  average  size  of  1  up  to  the  maximum  cluster  size  to 
be  expected  in  a  realistic  situation.  In  addition,  the  data  sets  must  include  scenarios 
in  which  reports  from  each  sensor  scan  interval  have  approximately  the  same  time 
stamps  and  scenarios  in  which  the  opposite  is  true. 


VI 


TESTS  OF  GATING  ALGORITHMS  FOR  TRACKING  OF 
MULTIPLE  OBJECTS  I.  THEORY,  REQUIREMENTS, 

AND  PERFORMANCE  MEASURES 

I.  Introduction 

The  purpose  of  this  paper  is  to  describe  measures  of  performance  and  data  set  require¬ 
ments  for  testing  algorithms  which  identify  and  select  candidate  associations  of  tracks  with 
sensor  data.  The  act  of  finding  these  candidate  associations  is  commonly  called  “gating” 
[2],  and  we  shall  use  that  term.  For  our  purposes,  the  sensor  data  come  in  the  form  of  “re¬ 
ports”  on  individual  objects  or  on  clusters  of  objects  which  cannot  be  resolved  separately. 
Tracking  algorithms  treat  such  clusters  as  single  objects  imtil  their  constituents  can  be 
resolved. 

Given  the  set  of  all  possible  pairings  of  an  individual  track  with  an  individual  report, 
a  gating  algorithm  selects  a  subset  containing  the  most  likely  pairings.  Each  pair  selected 
must  therefore  have  a  probability  of  association  that  exceeds  or  equals  some  minimum 
value  or  “threshold.”  In  this  way,  gating  reduces  the  number  of  candidate  pairs  that  must 
be  considered  in  constructing  tracks.  Ultimately  one  hopes  that  this  reduction  will  permit 
near-real-time  data  processing  and  tracking  of  the  objects  being  observed. 

Ovir  purpose  is  to  test  various  gating  algorithms  intended  for  tracking  large  numbers 
(10*  to  10*)  of  moving  objects.  The  algorithms  fall  imder  three  categories,  discussed  in 
the  subsections  below. 

I.l.  Brute  Force 

The  “brute  force”  approach  computes  a  likelihood  (probability)  or  “score”  for  eewdi 
possible  pair  and  selects  those  pairs  with  scores  that  equal  or  exceed  the  threshold.  This 
algorithm  should  find  all  such  pairs  and  thus  provides  a  baseline  for  determining  accmacy 
of  the  other  algorithms.  The  cost  equation  is 

C^f  =  NTNR{C^  +  C,-\-Ci)  .  (1.1) 

In  Eq.(l.l),  Nt  is  the  munber  of  tracks.  Nr  is  the  niimber  of  reports  received  during  a 
given  time  interval  (sensor  scan  interval),  Ce  is  the  cost  (in  central  processor  unit  seconds 
(CPU  s))  of  extrapolating  a  track  state  vector  and  covariance  matrix  to  the  time  of  a 
report,  C,  is  the  cost  of  computing  a  likelihood  or  score  for  one  track- report  pair,  and  Cd 
is  the  cost  of  comparing  two  data,  in  this  case  the  score  for  a  track-report  pair  and  the 
threshold  score,  in  order  to  select  the  candidate  pcdrs.  Notice  that  Eq.(l.l)  and  similar  cost 
relationships  in  Eq.(1.3)  and  Table  1  are  schematic  rather  than  exhaustive.  Our  intent  is  to 
reveal  the  underlying  scaling  of  the  various  algorithms  and  to  provide  a  basis  for  comparing 
the  overall  costs  of  corresponding  steps  in  the  algorithms. 
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In  Eq.(l.l),  assuming  that  Nt  and  Nr  are  both  proportional  to  the  actual  number  of 
objects  N,  Cbf  scales  as  N'^.  Given  present  computer  hardware,  the  brute  force  approach 
is  unacceptably  expensive  when  N  «  10*  or  10*.  We  note  that  the  various  per  item  costs 
might  be  far  from  equal  and  expect  that 

Ct>  Ce>  Cd  (  and  possibly  Ct'>  Ce>  Cd)  ,  (1.2) 

since  C,  requires  many  more  operations  than  track  extrapolation  or  simple  data  compari¬ 
son.  In  some  formulations,  the  score  calculation  could  implicitly  include  the  cost  of  traek 
extrapolation. 

1.2.  Nonhierarchical  Cluster  Algorithm  [1] 

This  algorithm  uses  spatial  clusters  that  have  been  identified  primarily  for  the  pur¬ 
pose  of  reducing  the  number  of  different  h}rpothese8  which  must  be  followed  in  a  multiple 
hypothesis  tracking  (MHT)  algorithm.  The  approximate  cost  may  be  computed  from  the 
steps  shown  in  Table  1.  To  reduce  the  scaling  from  brute  force,  one  first  averages  the  tracks 
within  a  cluster,  so  that  the  cluster  itself  may  be  treated  as  a  “supertrack.”  The  algorithm 
calculates  a  score  for  each  cluster-report  pairing.  The  extrapolation  of  the  cluster  (super¬ 
track)  to  the  time  of  the  report  is  implicit  in  the  algorithm.  The  scores  are  then  compared 
to  the  minimum  acceptable  value  or  threshold.  For  each  cluster  with  an  acceptable  score, 
the  algorithm  then  computes  a  score  for  each  constituent  track  and  again  compares  the 
score  to  the  threshold  to  find  the  candidate  trade-report  pairs.  In  Table  1,  C7a  is  the  average 
cost  of  computing  a  cluster  averaged  “supertrack”,  Nc  is  the  number  of  clusters,  N'q  is 
the  munber  of  selected  dusters  (usually  1  or  so),  and  Ntc  is  the  average  number  of  tracks 
per  duster.  Notice  that  if  each  duster  contains  one  track,  Nc  =  Nt  aud  the  scaling  of  the 
cost  is  NtNr  as  in  the  brute  force  approach.  However,  if  some  clusters  contain  more  than 
one  track,  then  Nc  <  Nt  aud  the  scaling  might  be  better  than  brute  force.  In  fact  the 
Appendix  shows  that  the  optimum  case  occurs  when  there  are  approximatdy  Nc  =  y/Nr 
dusters,  each  containing  on  average  y/Nr  tracks,  giving  a  cost  of 

Cc(min)«2(Ce-l-C,-l-Crf)N«v/}^  .  (1.3) 

Thus,  this  algorithm  scales,  at  best,  as  approximatdy  N^^^  and  at  worst,  as  N^.  The 
scaling  depends  primarily  on  the  track  and  report  data  and  on  the  scenario.  Obtaining 
N^/^  scaling  is  fortuitous  and  not  within  the  control  of  the  programmer.  As  indicated  at 
the  beginning  of  this  subsection,  clusters  are  important  in  the  generation  and  maintenance 
of  hypotheses.  The  case  of  Nc  «  ■\/Nt  and  Ntc  ^  y/^T^  while  optimiun  for  gating,  is 


not  optimum  for  hypothesis  generation  and  maintenance,  since  the  number  of  potential 
hypotheses  (or  scenes)  in  a  cluster  varies  as  (factorial).  In  addition,  if  y/N  exceeds 

the  maximiim  cluster  size  in  a  realistic  situation,  the  optimiim  case  for  gating  with  this 
algorithm  will  not  occur. 

Finally,  this  “cluster”  algorithm  shooild  not  be  confused  with  other  cluster  algorithms 
that  one  finds  in  the  computational  literature  [3,  4].  The  latter  are  hierarchical  and  scale 
as  NlogN.  For  this  reason,  we  view  the  above  nonhierarchical  cluster  algorithm  as  a 
sophisticated  brute  force  algorithm  worthy  of  consideration  because  of  its  simplicity  and 
potential  improvement  over  conventional  brute  force  techniques. 

1.3.  Near-Neighbor  Algorithms  [5]. 

Table  1  shows  the  costs  of  steps  used  in  a  typical  near-neighbor  algorithm  for  gating 
when  the  reports  correspond  to  different  observation  times  within  some  finite  time  interval 
(scan).  The  quantities  An  and  A,  are  constant  coefficients  of  order  1.  The  example  here 
is  that  of  projecting  the  tracks  forward  to  Mn  different  times  in  within  the  time  interval, 
where  Mn  is  a  small  number  (»  5)  [6].  For  each  report,  the  sJgorithm  identifies  the  closest 
data  structure  in  time  and  then  searches  it  for  all  tracks  falling  within  some  radius  of  tL  j 
position  of  the  report.  The  search  radius  depends  on  the  time  ijt  of  the  report,  the  time  in 
corresponding  to  the  data  structure  selected,  the  track  velocities,  and  the  threshold  score. 
Upon  finding  a  small  number  (denoted  Nn)  of  candidate  tracks,  the  algorithm  computes 
scores  and  makes  the  final  gating  decisions.  Notice  that  in  theory  this  algorithm  computes 
far  fewer  scores  than  the  cluster  algorithm,  and  thus  minimizes  the  most  costly  calculations. 
Setting  up  the  data  structures  scales  as  Nt  log  Nt,  but  the  costs  per  operation  are  very 
small  relative  to  C*. 

In  stimmary,  the  worst  scaling  part  should  be  superior  in  scaling  to  the  brute  force 
and  cluster  approaches.  The  near-neighbor  approach  offers  the  additional  advantage  that 
the  poorest  scaling  step  does  not  involve  a  score  calculation.  These  are  the  features  that 
have  led  to  massive  savings  in  molecular  dynamics  calculations  and  have  been  the  main 
selling  points  for  the  present  application. 

1.4.  Further  Comments 

The  above  discussion  indicates  the  following  potential  advantages  for  near-neighbor 
algorithms: 

(1)  The  inherent  scaling  of  the  near-neighbor  algorithms  should  be  superior  by  a  factor 
of  N/  log  N  relative  to  brute  force  and  by  a  factor  of  at  least  \/N  /  log  N  over  the  cluster 
algorithm  and  perhaps  closer  to  N/  log  N  . 


TABLE  I 

Cost  of  Steps  in  Association 


Si  ^ 


<d 

%  II 


w  u 
G  9 

O  i? 
U  "S 


j2  w 
«0  0) 

«  £ 

-♦J 

«J  ■»' 

U  C) 
X  9 

W  I 

r  9 

hO  h 


o  o 


X  "iro 
oo  o 


aw 

9 

h  ^ 

o  I 


CO 

S  £ 
8  ® 
w  ts 

®  .H 

9  " 

c"  ^ 
S  V 

o  "5 
U  OT 


*3  O 
a  V 

lb“ 


9  Ji 
eo 


4 


(2)  The  most  expensive  calculation,  that  of  gating  scores,  scales  as  for  brute  force  zind 
between  and  for  the  cluster  algorithm.  The  same  step  scales  only  linearly  with 
N  in  the  current  implementation  of  the  near-neighbor  algorithms. 

(3)  In  fact,  the  number  of  score  calculations  per  report  remains  relatively  constant  with  N 
in  the  case  of  near-neighbor  algorithms  but  increases  as  N  and  at  least  v/77  in  the  other 
cases. 

The  purpose  of  the  present  tests  are  twofold: 

(1)  To  determine  the  expense  of  the  above  algorithms  in  performing  gating.  This  addresses 
directly  the  validity  of  the  above  theoretical  analysis  and  the  accompanying  conclusions 
for  systems  containing  large  numbers  of  objects.  In  Section  II,  we  consider  scaling  and 
actual  cost  separately. 

(2)  To  determine  the  accuracy  of  the  above  algorithms  in  performing  gating  (see  Section 
II.2).  In  other  words,  what  percentage  of  the  candidate  pairs  with  scores  above  threshold 
do  the  algorithms  find?  An  alternative  form  of  this  question  is,  “For  each  report,  what 
percentage  of  the  Ng  or  fewer  highest-valued  tracks  with  scores  exceeding  the  threshold 
does  the  algorithm  find?”  Here  Ng  is  the  limit  usually  set  by  a  gating  algorithm  on  the 
number  of  tracks  which  may  be  associated  with  each  report  and  depends  on  some  con¬ 
straint,  such  as  the  required  speed  or  available  computer  memory.  Another  approach  is  to 
weight  this  percentage  by  the  scores  of  the  pairs  fo\md,  so  that  missing  lower- valued  pairs 
is  less  important  [7].  As  indicated  by  the  form  of  the  second  question,  the  possibility  exists 
that,  for  a  given  report,  fewer  than  Ng  tracks  have  scores  exceeding  the  threshold  (Section 
II.2). 

Two  sources  of  data  and  algorithms  are  available  for  the  tests.  First,  a  realistic 
Tracker- Correlator  (TRC)  Code  [1]  using  the  above  cluster  algorithm  can  provide  data 
on  tracks  and  reports,  and  the  portion  of  actual  code  dealing  with  the  cluster  algorithm 
could  be  part  of  the  overall  test  package.  The  available  TRC  can  comfortably  handle  sets 
of  order  10  to  100  objects  and  not  many  more  than  1000  objects.  For  that  reason,  the 
authors  plan  to  conduct  tests  on  the  three  algorithms  using  simpler,  “simxilated”  tracks 
and  reports  numbering  up  to  6.4  xlO*,  which  is  more  representative  of  possible  scenarios. 
The  authors  plan  to  write  the  computer  code  for  all  three  algorithms  The  tests  will  take 
place  on  a  SUN  workstation. 

The  next  two  sections  of  this  paper  deal  with  the  data  set  requirements  and  the 
appropriate  measures  of  performance  for  the  tests.  The  fourth  section  identifies  other 
important  parameters  and  factors  in  the  definition  of  data  sets. 
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II.  Data  Requirements 

As  indicated  above,  the  tests  will  address  two  criteria  for  selection  of  a  gating  algo¬ 
rithm:  expense  and  accuracy.  The  data  set  requirements  for  determining  these  two  features 
might  be  different.  We  will  consider  them  in  turn. 

II.  1.  Cost  Measurement 

Cost  or  expense  involves  (1)  the  various  costs  per  operation,  denoted  C,,  Ce,  etc. 
in  the  previous  section,  and  (2)  the  scaling  of  the  algorithms  with  the  number  of  objects 
N.  Consider  scaling  first.  Given  reasonable  computer  programming  techniques  and  within 
a  computer  hardware  class,  the  scaling  of  the  algorithms  (or  their  major  steps)  with  N 
should  be  reasonably  independent  of  the  particular  machine  used  and  of  the  quality  of 
the  actual  programming.  Thus,  scaling  with  N  is  one  aspect  which  these  tests  should 
determine  with  reasonable  generality. 

The  actual  cost  in  terms  of  CPU  seconds,  however,  will  also  depend  on  the  costs  per 
operation  of  the  particulsu:  computer  code  on  the  particular  machine  and  will  depend  on 
the  quality  of  programming.  The  measurement  of  CPU  time  is  important  when  one  is 
seeking  the  maximum  speed  of  a  given  algorithm  on  a  specific  machine. 

For  the  present  tests,  since  absolute  speed  is  not  the  goal  of  the  overall  project, 
scaling  with  N  should  be  the  most  important  aspect  of  cost  to  measure.  To  accomplish 
this  requires  performance  data  for  different  scenarios,  individually  containing  different 
nximbers  of  objects.  To  accoxmt  for  situations  which  are  more  favorable  or  less  favorable 
to  the  cluster  algorithm,  the  scenarios  shoxild  also  cover  the  range  of  clustering  from  an 
average  size  of  1  to  the  maximum  number  of  objects  in  a  cluster  for  a  realistic 

situation. 

Let  N~  and  denote  the  minimum  and  maximum  values  of  N  covered  by  the  set 
of  scenarios.  What  are  the  smallest  acceptable  values  of  these  numbers?  Given  scenarios 
with  similar  distributions  of  clusters  and  cluster  sizes,  the  minimum  value  iV’~sho\ild  be 
several  times  the  maximum  number  of  objects  per  cluster.  Alternatively,  N”  should  be 
several  times  Ng,  the  maximum  allowed  number  of  tracks  selected  by  the  gating  algorithm 
for  each  report.  In  either  case,  the  gating  algorithm  should  then  eliminate  a  significant 
number  tracks  from  consideration  for  pairing  with  any  report  and  a  trivial  result  will  be 
avoided.  In  the  scenarios  presently  envisioned,  the  maximum  number  of  objects  in  a  cluster 
will  be  of  order  10,  as  will  Ng.  In  that  case,  we  expect  that 

N-  >  100  .  (2.1) 
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Now  consider  N'^.  To  determine  scaling  for  applications  to  systems  with  large  niunbers 
of  objects,  say  N  >  10^,  iV"*" should  at  least  be  10^,  so  that  extrapolation  to  the  next  power 
of  10  or  so  will  be  as  believable  as  possible.  In  addition,  one  would  like  to  use  >  lOON" 
to  meike  sure  that  the  scaling  functionality  is  stable  over  £in  appreciable  range  of  values  for 
N.  Equation  (2.1)  then  gives  us  >  10*. 

We  can  also  ask  a  related  question  about  “For  what  value  will  the  range  (N~ , 
N'^)  be  large  enough  to  discriminate  the  best  possible  scaling  of  the  nonhierarchical  cluster 
algorithm  from  the  worst  scaling  claimed  for  the  near-neighbor  algorithms?”  Thus  we  must 
distinguish  from  NlogN.  At  this  point  we  caimot  estimate  the  scatter  in  the  data 
over  the  range  of  scenarios,  but  we  can  compute  the  differences  which  will  arise  for  various 
values  of  JV“and  .  As  a  measure,  we  compute  the  ratio  of  maximum  to  minimtun  costs 
for  the  two  scaling  functions  and  divide  one  by  the  other: 


logN- 

v^'^logiV+  • 


(2.2) 


Fiirure  1  depicts  the  meaning  of  TZ  in  terms  of  a  graph  of  cost  C  versus  N.  To  compzire 
the  two  different  algorithms,  we  normalize  the  respective  cost  values  within  the  range 
(iV“,  N'^)  to  the  values  at  IV  =  N~  (see  Eq.(3.9)).  Then  C  starts  out  at  a  value  of  1 
in  both  cases.  At  N  —  the  ratio  of  normalized  costs  for  the  two  functions  will  be 
TZ{N~  Thxis  TZ.{N~ ,N'^)  gives  the  maximum  deviation  of  the  two  functions.  Now 

define  'R,(N~,N)  to  be  the  ratio  of  the  normalized  values  of  the  two  functions  for  any 
N.  This  we  obtain  by  replacing  iV^'by  N  in  Eq.(2.2).  As  N  decreases  in  value  from 
TZ{N~  ,N)  approaches  1,  meaning  that  the  differences  between  the  functions  become  less 
easy  to  discriminate. 

Now  choose  iV~=  100,  the  lowest  value  indicated  in  Eq.(2.1).  For  N  =  iV*"  =  1000, 
we  have  72  «  2.  Unless  the  scatter  in  the  data  are  too  large,  this  should  be  enough  to  dis¬ 
tinguish  that  a  scaling  difference  exists  between  the  cluster  and  near-neighbor  approaches. 
Since  the  differences  decrease  with  JV,  the  range  of  N  from  100  to  1000  might  be  marginal 
for  determining  the  actual  scaling  functionality  of  the  two  approaches.  At  TV  =  TV+  =  10*, 
we  obtain  72  =  5,  and  for  N  =  JV*"  =  6  x  10*,  we  obtain  72  =  10.  Thus  to  determine  the 
actual  scaling  functionality,  scenarios  ranging  from  N  =  100  to  TV  =  10*  or  6  x  10*  seem 
to  be  much  better. 

II.2  Accuracy  Determination 

Now  consider  the  question  of  accuracy,  and  define  the  ideal  result  as  follows.  For  a 
given  report  r,  Let  TVj;  be  the  number  of  tracks  with  scores  equal  to  or  exceeding  the 
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Figure  1.  Log-Log  plot  of  normalized  cost  Eq.(3.9)  versus  niunber  of  objects  N  for  costs 
varying  as  and  N  log  N.  Since  the  scales  are  logarithmic,  the  distances  marked  with 
arrows  are  proportional  to  the  logarithms  of  the  ratio  7l(^N~,N)  of  the  two  normalized 
cost  functions  for  JV  =  10’  and  10^,  respectively. 


threshold.  Remember  2dso  that  the  gating  algorithm  usually  selects  no  more  than  Ng 
tracks  per  report.  Then  if  <  Ng,  the  ideal  result  is  to  identify  all  Nxr  pairings  of 
tracks  with  report  r.  If  Nt,.  >  Ng,  the  ideal  result  is  to  identify  the  Ng  highest- valued 
pairs.  The  performance  of  the  various  algorithms  should  be  measured  against  the  idezd 
result  for  each  scenario  considered.  One  might  wish  to  vary  the  threshold  to  determine  its 
effects  on  the  process. 

Once  again,  we  desire  a  set  of  scenarios  covering  a  range  (N~,  N'^)  of  values  for  N, 
the  number  of  objects.  Equation  (2.1)  limits  N~  to  a  ■v'alue  of  100  or  greater  to  ensure 
a  nontrivial  result.  However,  for  the  determination  of  accuracy,  a  value  of  iV"*"  =  6  x  10* 
is  potentially  too  high.  The  two  most  important  factors  in  accuracy  appear  to  be  the 
degree  of  clustering  inherent  in  each  scenario  and  the  variation  of  observation  times  (time 
“stamps”)  of  the  reports  gathered  during  each  sensor  scan  interval.  A  scenario  with  little 
spatial  clustering  of  the  moving  objects  (separations  exceeding  sensor  resolution)  and  in 
which  new  batches  of  reports  have  approximately  the  same  time  stamps  might  be  better 
for  near-neighbor  algorithms  than  a  scenario  having  the  opposite  properties  (see  Section 
IV.2).  We  expect  these  factors  to  have  less  of  an  influence  on  the  accuracy  of  the  cluster 
algorithm. 

Of  the  two  factors,  only  clustering  should  affect  the  value  of  N"^ .  To  cover  the  range 
of  possibilities,  each  scenario  should  contain  clusters  with  an  average  size  somewhere  in 
the  range  from  1  to  N^^,  the  maximum  cluster  size  to  be  expected  in  any  scenario.  To 
insure  a  nontrivial  result,  many  more  than  one  cluster  should  be  present.  This  leads  us 
to  propose  that  N"^  be  greater  than  by  a  factor  of  at  least  10.  In  addition,  to 

cover  a  reasonable  range  of  values  for  N,  iV'*' should  be  greater  than  N~  by  a  factor  a 
least  10.  Since  N~  >  100  and  N^^  is  of  order  10,  these  considerations  indicate  that 
iV'*'  >  1000.  Thus  scenarios  covering  the  range  from  100  to  1000  objects  should  be  useful 
for  determining  accuracy  of  the  algorithms. 

II.3.  Summary  of  Data  Requirements 

For  measuring  expense  of  the  algorithms,  the  tests  should  emphasize  scaling  with  the 
munber  of  objects  N.  The  data  sets  should  consist  of  various  scenarios  covering  a  range 
100  <  IV  <  6  X  10*.  The  data  sets  currently  within  the  capability  of  the  realistic  correlator- 
tracker  code  will  not  satisfy  these  requirements  at  the  upper  end  of  the  range  and  could 
be  marginal  for  determining  scaling. 

For  measuring  accmacy  of  the  algorithms  and  to  cover  the  situations  which  affect 
the  scaling  of  the  cluster  algorithm,  the  scenarios  should  cover  a  range  in  average  cluster 
size  from  a  value  of  1  up  to  the  maximiun  size  expected  in  realistic  situations.  Data  sets 
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covering  100  <  N  <  1000  shovdd  be  useful  and  perhaps  adequate  for  measuring  accuracy. 
In  addition,  the  data  sets  must  include  scenEirios  in  which  reports  from  each  sensor  scan 
interval  have  approximately  the  same  time  stamps  and  scenarios  in  which  the  opposite  is 
true. 

III.  Measures  of  Performance 
III.l  Accuracy 

We  now  identify  measmes  of  performance  which  seem  to  be  appropriate  for  these  tests. 
For  accuracy  the  baseline  will  be  the  “ideal  result”  defined  in  the  previous  section.  Given 
that  the  classic  brute  force  approach  cbmputes  a  score  for  each  possible  pair  and  compares 
the  scores  to  the  threshold  value,  the  result  of  the  brute  force  algorithm  for  each  report  will 
be  identical  with  the  ideal  result.  Denote  by  X  the  set  of  all  track-report  pairs  constituting 
the  ideal  resxilt  for  the  entire  set  of  reports.  Then  if  Ah  is  the  set  of  pairs  identified  by 
the  brute  force  algorithm,  we  have  X  =  A*.  We  must  measure  the  intersection  of  X  with 
the  sets  Ac  and  An  of  pairs  actually  found  by  the  cluster  and  near-neighbor  algorithms, 
respectively.  Thus  the  performance  V  of  the  near-neighbor  algorithm  relative  to  accuracy 
is 


M(Annx) 
-  M(X) 


(3.1) 


where  M  denotes  a  measure  function.  A  similar  equation  holds  for  the  cluster  algorithm. 
Two  performance  measures  have  been  mentioned  previoiisly  [7]  but  not  specified  in  an 
equation: 

(1)  Define  Afi(5)  to  be  the  number  of  elements  in  a  set  5.  Compute  the  first  performance 
measxire  Vi  as  the  percentage  of  pairs  in  X  which  are  also  contained  in  the  set  A  of  pairs 
fo\md  by  a  near-neighbor  or  cluster  algorithm, 


^  MijXnA) 

“  Mi(X) 


(3.2) 


(2)  For  a  set  S,  define  Al2(5)  as  the  product  of  M.i{S)  and  the  sum  of  scores  in  S.  This 
gives  us  a  second  performance  measure,  equivalent  to  multiplying  Vi  by  the  ratio  of  the 
sums  of  scores  for  the  elements  in  the  sets  AdX  and  the  ideal  set  X, 


V2=  “Pi  X 


J 


(3.3) 
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In  Eq.(3.3),  the  score  or  likelihood  for  the  pth  pair  is  Sp.  The  second  measure  V2  has  the 
advantage  that  the  penalty  for  missing  a  lower-valued  pair  in  J  is  less  than  for  missing 
a  higher-valued  pair.  Thus  the  relative  importance  of  the  various  pairs  comes  into  the 
measure  of  performance. 

A  variation  on  the  above  approach  is  possible.  Denote  by  Ir  the  set  of  track-report 
pairs  in  the  ideal  result  for  a  given  report  r: 

J,=  |(t,r)|t€Tand(<,r)6j|  ,  (3.4) 

where  T  is  the  set  of  all  tracks.  Thus  Jr  is  the  ideal  result  for  a  specific  report  r,  and  we 
have 

I  =  il^r  .  (3.5a) 

r 

Similar  equations  hold  for  the  sets  Aer  and  Anr  of  tracks  identified  by  the  cluster  and 
near-neighbor  gating  algorithms  for  a  report  r: 

r 

=  (3.5c) 


We  rATi  now  compute  the  above  performance  measures  separately  for  each  report  r 
and  average  them  over  all  reports.  For  report  r  and  the  set  Ar  of  tracks  identified  by  the 
cluster  or  near-neighbor  gating  algorithm,  define  the  performance  measures  to  be 


and 


_MiiIrnAr) 
■  Miilr) 


VrO,  —  Vrl  X 


(3.6) 

(3.7) 


Then  the  overall  performance  measures  corresponding  to  and  V2  are  the  averages 
of  Vri  and  Vr^  over  all  reports: 
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=  ’  =  1’  2-  (3-8) 

Experience  indicates  that  the  measures  averaged  over  reports  "Pj  and  Pj  might  deviate  from 
the  value  1.0  more  than  the  measures  for  the  total  set  of  reports  Pj  and  Pj.  However,  the 
corresponding  measures  should  be  quite  similar.  Note  also  that  Eqs.(3.6)  and  (3.7)  assume 
that  Jr  ^  0,  the  null  set. 

III.2.  Scaling 

For  seeding  we  must  compute  the  cost  of  ea,ch  step  in  the  gating  process,  as  identified 
in  Table  1,  for  each  scenario.  The  set  of  scenarios  will  cover  a  range  of  values  {N~,  N'^)  of 
the  number  of  objects  N.  The  most  convenient  unit  of  cost  to  measure  is  central  processor 
unit  (CPU)  seconds.  By  treating  each  step  of  gating  separately,  we  will  be  able  to  isolate 
the  portions  of  the  gating  process  with  different  scaling  with  N,  as  indicated  by  Table  1. 
To  eliminate  programming  differences  between  the  various  candidate  algorithms,  we  will 
normalize  the  cost  to  the  value  at  N  =  N~: 


Thus  the  lowest  value  of  C  for  each  of  the  algorithms  will  be  1. 

Once  sufficient  cases  have  been  nm  to  obtun  costs  at  five  or  more  values  of  N  in  the 
range  (N~ ,  N"^),  we  will  fit  the  functions  log(Cc)  (cluster  algorithm)  and  log(Cj,)  (near¬ 
neighbor  algorithm)  to  the  function  f{N)  =  a  log(iV)  -|-  The  quantity  a  is  the  slope 
of  the  curve  and  measures  the  power  of  N  with  which  a  step  varies.  Thus,  a  should  be 
2  for  the  brute  force  algorithm  and  fall  between  2  and  3/2  for  the  cluster  algorithm.  For 
those  steps  which  might  scale  as  JVlogiV,  we  will  also  fit  the  function  C'N~/N  to  f{N) 
to  determine  whether  the  slope  is  constant  and  equal  to  iog(V-j  range  of  values 

for  N.  The  accuracy  of  the  results  will  depend  critically  on  the  scatter  in  the  data,  which 
we  cannot  estimate  at  present. 

IV.  Further  Considerations 

Several  additional  factors  could  have  measurable  effects  on  the  resvdts,  and  taking 
them  into  account  could  increase  the  number  of  different  scenarios  necessary  for  the  tests. 
Ancillary  data  from  the  tests  might  also  be  useful  to  designers  of  operational  systems  in 
the  future.  We  will  briefly  discuss  these  ideas  below. 
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IV. 1  Types  of  Sensors 

The  two  main  types  of  sensors  to  be  considered  are  active  (radar)  and  passive  (e.g., 
infr2ired  (IR)  or  electro-optical  (EO)).  The  radar  data  are  three-dimensional  while  the  pas¬ 
sive  data  come  in  the  form  of  bearings  (two  angles)  and  are  effectively  two-dimensional. 
The  two  types  of  data  represent  different  programming  problems  for  near-neighbor  algo¬ 
rithms.  We  plan  to  concentrate  on  radar  data  at  the  outset  and  treat  the  IR  problem  in 
the  second  stage  of  the  tests. 

rV.2.  Temporal  Distribution  of  Data 

Two  possibilities  for  scenarios  exist  [5].  Either  the  observations  by  a  sensor  during  a 
scan  all  correspond  to  approximately  the  same  time  (i.e.,  the  scan  period  is  short  relative 
to  the  time  scale  on  which  the  objects  change  position  significantly),  or  the  observations 
are  distributed  over  a  nonnegligible  scan  time.  The  former  case  is  advantageous  for  near¬ 
neighbor  algorithms,  since  the  track  file  data  structure  must  be  set  up  only  once  per  scan 
and  the  reports  all  have  roughly  the  same  time  “stamp”  as  the  tracks,  making  calculation 
of  scores  easier.  The  latter  case  is  more  difficult  because  the  observation  times  are  different 
and  candidate  tracks  must  be  projected  to  each  of  those  times  to  compute  the  respective 
scores.  Table  1  presents  a  near-neighbor  method  imder  the  assumption  of  the  latter  case. 
Presumably  both  situations  should  be  addressed  by  the  tests. 

IV.3.  Choice  of  Threshold  Score 

The  threshold  score  for  determining  the  acceptable  track-report  pairings  will  deter¬ 
mine  the  average  nvunber  of  tracks  selected  by  the  gating  algorithm  per  report.  This  will 
also  determine  the  maximum  number  of  hypotheses  which  could  be  created.  As  the  thresh¬ 
old  score  increases,  the  accuracy  of  the  near-neighbor  algorithms  should  also  increase,  since 
the  acceptable  tracks  must  be  closer  to  each  report.  In  this  case,  we  should  reach  the  sit¬ 
uation  of  having  to  investigate  only  the  nearest  one  or  two  tracks  for  each  report.  The 
situation  in  which  reports  have  different  observation  times  shoffid  also  be  easier  when  the 
threshold  score  is  higher. 

IV.4.  Number  of  Different  Scenarios 

The  data  set  for  each  N  should  include  a  number  of  different  scenarios  for  each  value 
of  the  average  cluster  size  and  each  representative  sensor  scan  interval.  In  addition  the 
data  sets  must  cover  several  different  values  of  N.  We  see  in  this  the  possibility  of  another 
“combinatorial  explosion”  and  will  have  to  determine  the  minimal  number  of  data  sets 
necessary  during  the  course  of  the  tests.  The  basis  for  this  decision  will  likely  be  the 
scatter  observed  in  the  results  during  the  initial  testing  phase. 
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rV.5.  Coefficients  of  Scaling  Factors 

As  indicated  previously,  the  tests  should  emphasize  scaling  and  not  absolute  speed. 
However,  from  the  type  of  hardware  used  for  the  tests  and  the  actual  CPU  time  expen¬ 
ditures  for  the  gating  tests,  one  can  often  estimate  speed  on  other  systems.  Thus,  even  if 
the  software  is  not  optimized  and  even  though  speed  is  hardware  dependent,  elapsed  CPU 
time  can  be  useful  to  system  designers  who  wish  to  employ  the  algorithms  that  we  have 
developed.  While  we  will  not  present  such  data  as  part  of  the  test  results,  we  can  use  the 
data  to  estimate  speed  of  near-neighbor  algorithms  on  other  machines,  if  so  requested. 
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Appendix 

Optimal  Scaling  of  the  Nonhierarchical  Cluster  Algorithm 


This  appendix  deals  in  more  detail  with  the  costs  of  the  nonhierarchical  cluster  algo¬ 
rithm  and  presents  an  approximate  equation  for  the  optimal  scaling  of  cost  with  N,  the 
number  of  objects  in  the  scenario.  First,  the  equations  given  in  Step  4  of  Table  1  require 
further  explanation.  The  computation  of  scores  for  all  tracks  in  the  clusters  that  have  been 
selected  for  association  with  report  r  incurs  the  following  expense: 

^4r  =  (C',  +  Ce)  .  (A.1) 

{ch 

Here  C^r  is  the  cost  of  step  4  (Table  1)  for  report  r;  {c}r  denotes  the  set  of  clusters  which 
have  scores  that  exceed  the  threshold  for  association  with  report  r;  and  Nt^  is  the  number 
of  tracks  in  cluster  c.  Section  I  defines  the  remaining  quantities.  We  now  define  the  bracket 
notation 

•  {•) 

for  an  average  over  a  ntunber  Na  of  quantities  which  are  labelled  by  the  set  of  indices  {a}, 
not  necessarily  equal  to  {1, 2, . . . ,  iV*}.  Averages  of  different  sets  will  use  the  same  bracket 
notation,  even  though  the  sets  may  have  different  nxunbers  of  elements.  Further,  if  the 
index  set  {a}  depends  on  another  index  i,  i.e.,  {o}  =  {o},-  then  the  right  bracket  will  have 
a  subscript  i,  in  other  words,  (  )<.  If  the  set  {a}  =  {1,2, ...,iVa},  then  we  will  use  the 
usual  notation  for  the  stun  and  need  not  carry  the  subscript  «  within  the  brackets: 

if.)  =  if)  =  ■ 

*  a»l 

The  total  cost  of  computing  scores  and  determining  which  tracks  should  be  associated 
with  each  report  in  the  set  of  data  is  then 
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C4  =  (C.  +  C.  +  Crf) 

r=l  {c}, 

=  (C,  +  C.  +  C.)  ^  ATJ,,,  ( ^  Nr, 

r=l  V-^cCr)/  {,j. 

Nr 

=  iC.+C,  +  Cu)  Yl^<r){NTjr 

rs=l 

=  (C,  +  C,  +  Cj)  i^«(Wj(,)(JVT,)r> 


(A.2) 


The  last  line  of  Eq.(  A.2)  shows  the  average  over  reports  of  the  product  of  two  quantities:  (a) 
the  number  of  clusters  selected  for  each  report  r  and  (b)  the  average  over  those  particular 
clusters  of  the  number  of  tracks  in  a  cluster.  The  superscript  '  differentiates  the  niunber 
of  clusters  selected  for  a  report  from  the  total  number  of  clusters  Nc-  To  obtain  the 
relationship  shown  in  Table  1,  Step  4,  one  can  assume  either  that  the  number  of  clusters 
exceeding  the  threshold  score  is  approximately  the  same  for  each  report  or  the  number  of 
tracks  per  clijister  is  approximately  the  same.  Equation  (A.2)  then  becomes 


-- Nr[N',){Ntc)  , 


(A.3) 


in  which  we  have  left  the  subscript  “r”  off  of  the  average  quantities  for  convenience.  For 
presentation  of  this  equation  in  Table  1,  we  define  (iV’r*)  =  Ntc  and  (iV')  s  Nq.  Firom 
Table  1,  we  see  that  the  total  cost  for  the  cluster  algorithm  is  then 


Cc  =  NrCa  +  NciCc  +  C,+  C4)Nr  +  N'cNtc{C,  +  C,  +  Ch)Nr  .  (A.4) 


Notice  that  Ntc,  and  Nc  are  related  through  the  equation 

iVTC  =  ~  .  (A.5) 

Nc 

To  find  an  extremum  in  the  cost  relative  to  the  number  of  clusters,  one  substitutes  Eq.(A.5) 
into  Eq.(A.4),  computes  the  partial  derivative  of  the  result  with  respect  to  Nc  and  sets 
the  result  to  zero  to  obtain 


N’cNt  =  NI 


(A.6) 
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An  additional  partial  derivative  with  respect  to  Nc  is  greater  than  zero  when  Eq.(A.6) 
holds,  showing  that  the  cost  is  minimized  by  Eq.(A.6).  The  resulting  cost  equation  is 


Ccmin  =  NrCa  +  2NrJNtN'c{C,  +  C.  +  C^) 

(A.7) 

To  obtain  the  second  hne  of  Eq.(A.7),  we  have  assmned  that  the  cluster  averaging  (Step  1, 
Table  1)  costs  far  less  than  the  score  calculation  and  that  N'q  w  1.  Also,  for  large  numbers 
of  objects  the  term  with  the  worst  scaling  should  dominate. 


«  2NRy/N^iCe  +  C,  +  Ci) 


